Deep learning-based quantitative analyses of spontaneous movements and their association with early neurological development in preterm infants

This study aimed to develop quantitative assessments of spontaneous movements in high-risk preterm infants based on a deep learning algorithm. Video images of spontaneous movements were recorded in very preterm infants at the term-equivalent age. The Hammersmith Infant Neurological Examination (HINE) was performed in infants at 4 months of corrected age. Joint positional data were extracted using a pretrained pose-estimation model. Complexity and similarity indices of joint angle and angular velocity in terms of sample entropy and Pearson correlation coefficient were compared between the infants with HINE < 60 and ≥ 60. Video images of spontaneous movements were recorded in 65 preterm infants at term-equivalent age. Complexity indices of joint angles and angular velocities differed between the infants with HINE < 60 and ≥ 60 and correlated positively with HINE scores in most of the joints at the upper and lower extremities (p < 0.05). Similarity indices between each joint angle or joint angular velocity did not differ between the two groups in most of the joints at the upper and lower extremities. Quantitative assessments of spontaneous movements in preterm infants are feasible using a deep learning algorithm and sample entropy. The results indicated that complexity indices of joint movements at both the upper and lower extremities can be potential candidates for detecting developmental outcomes in preterm infants.


Scientific Reports
| (2022) 12:3138 | https://doi.org/10.1038/s41598-022-07139-x www.nature.com/scientificreports/ assessments of spontaneous infantile movement, it is important to quantitatively analyze the kinematics of the upper and lower extremities in preterm infants. There have been previous attempts to analyze the spontaneous infantile movements based on automatic assessments. The automatic assessments can be helpful to reduce reliance on assessors, thereby decreasing human error. The sensing modalities of the automatic assessments in previous studies can be categorized into indirect sensing, where hardware is placed around the assessment environment (e.g. RGB cameras, 3D motion capture, and Microsoft Kinect) and direct sensing, where movements are captured using hardware attached to bodies (e.g. inertial sensors and magnet tracking systems) 17 . Particularly, the vision-based approaches using RGB cameras has been getting great interest with popularization of smartphones and recent advances in pose-estimation models based on deep learning algorithms. This approach can have several advantages compared to methods using direct sensing modalities including relative easiness to understand, high spatial resolution, high context information, non-intrusiveness, not-dependence on reflective markers or inertial sensors, and high availability 17,18 .
Previous researches using vision-based approaches have been steadily published for the past two decades 16,19 . To acquire data by tracking the location of the specific body structures, a motion tracking system using reflective markers 20,21 were used and much simple methods using a single RGB camera have been adopted in recent studies 18,[22][23][24][25] . This can be attributed to the progress of automatic pose-estimation techniques for two-dimensional image data and their disclosure as open source models including OpenPose 26 and AlphaPose 27 . The previous studies have usually aimed to establish methods to quantify and classify spontaneous infantile movements [22][23][24][25]28 or to develop prediction models to estimate developmental delay or cerebral palsy 15,18,20 . Although the classification or prediction performance of the previous models was relatively satisfactory, these models may provide information based on engineering techniques that can be unfamiliar or not be straightforward to clinicians to interpret and utilize for clinical decision-making. Clinicians may require more easily interpretable results based on quantitative assessments of important spatiotemporal variables.
Therefore, this study aimed to develop automatic standardized methods for quantitatively analyzing spontaneous movements in preterm infants and to analyze spontaneous movements according to early neurological development based on established quantitative assessments.

Methods
Participants. From March 2019 to January 2020, preterm infants with gestational age < 32 weeks or birth weight < 1500 g admitted to the neonatal intensive care units of two tertiary hospitals were enrolled in this study. Infants were excluded if they had genetic syndromes, major congenital malformations, or medically unstable conditions, such as requiring cardiovascular support, active sepsis, or any major surgery that could affect the spontaneous infantile movements. Informed consent was obtained from parents or legal guardians of all infants according to the institutional guidelines. After discharge, the participants entered the standard followup programs, including pediatric and neurologic evaluations and parental educational support. The study was approved by the institutional review boards of Seoul National University Hospital and Chung-Ang University Hospital. It was performed in accordance with all relevant guidelines and regulations.

Clinical information.
Clinical information of the infants at the perinatal and postnatal periods was obtained including infantile characteristics: sex, gestational age, postnatal age, birth weight, global score of HINE, corrected age at HINE, categories of GMA; neonatal morbidities: 1-and 5-min Apgar scores, bronchopulmonary dysplasia severity 29 , antenatal steroid use, periventricular leukomalacia, intraventricular hemorrhage, period of invasive ventilator use, patent ductus arteriosus, treatments of patent ductus arteriosus, retinopathy of prematurity, history of sepsis, and history of seizure; and maternal characteristics: maternal preeclampsia, gestational diabetes mellitus, multiple gestations, and maternal chorioamnionitis. Categories of GMA were determined by one certified expert who were blinded to the clinical history of the infants. Brain magnetic resonance imaging at term-equivalent age was performed in infants with birth weight < 1000 g, gestational age < 29 weeks, or those with severe intraventricular hemorrhage on brain ultrasonography.

Video recordings of spontaneous movements. Video images of spontaneous movements in preterm
infants were recorded according to the standard protocol of GMA at term-equivalent age (40 ± 1 weeks' postmenstrual age) 8 . The instructions for making video recordings of spontaneous movements in this study were as follows: infants are (1) lying supine, naked, fully awakened, but without agitation, such as crying or fussing; (2) in a comfortable, quiet, and undisturbed condition at a neutral temperature; (3) free to move their bodies and all limbs including the fingers and toes, which are included in the recording scenes; and (4) recorded using a conventional or smartphone RGB camera 30,31 for 3-5 min with at least 2 continuous min recorded by a research assistant with a certificate in occupational therapy or the parents. In cases of discharge from the hospital before the term-equivalent age, the parents were requested to record and transmit videos of their infants at home to the research team using their smartphone camera. The process of video recordings and transmission was explained to the parents or legal guardians before and after hospital discharge. If the recorded videos were considered unsuitable according to the instructions, recordings were conducted again within the period of term-equivalent age. The recorded videos were edited and combined into 3-5-min sequences.
Automated movement recognition and kinematic analysis. The positional coordinates of twelve joints-the bilateral shoulders, elbows, wrists, hips, knees, and ankles-were automatically extracted from the recorded videos of spontaneous movements in the preterm infants using a pose-estimation algorithm, Alpha-Pose (Fig. 1A,B). This is a pretrained pose-estimation model developed based on a convolutional neural network architecture 27 . It can produce positional coordinates of each joint with confidence levels from 0 to 1 at www.nature.com/scientificreports/ each frame. The positional coordinates with confidence levels < 0.5 were regarded as measurement errors and eliminated from the positional data. Joint angles and joint angular velocities were determined as basic data in further kinematic analyses because these parameters are less susceptible to differences in camera distance from or angle of viewing of the infants compared to positional coordinates per se or the length between adjacent joints. The joint angle was defined as the angle formed by the positional coordinates of the corresponding joint and its two adjacent joints (Fig. 1C). The shoulder joint angle was defined as the angle made by the corresponding shoulder joint, contralateral shoulder joint, and ipsilateral elbow joint; the elbow joint angle is made by the corresponding elbow joint and the adjacent shoulder and wrist joints; the hip joint angle is made by the corresponding hip joint, contralateral hip joint, and ipsilateral knee joint; and the knee joint angle is made by the corresponding knee joint and the adjacent hip and ankle joints. The data of joint angles were acquired frame by frame and were interpolated with a locally weighted smoothing method for processing missing data and noise reduction. Joint angular velocities were approximated using the symmetric difference quotient as the sequence of the finite differences of the joint angles. After preprocessing of time-series data of joint angles and joint angular velocities, the maximum, minimum, mean, and standard deviation values were calculated for each infant. Inter-limb synchronization and complexity during spontaneous movements were investigated with kinematic analysis since these are hallmarks of brain dysfunction and are associated with developmental disorders 32 . To quantify complexity of limb movements in preterm infants, this study adopted sample entropy (SE), a measure of the degree of signal regularity for timeseries data as a complexity index 33 . Sample entropy is a measure of the degree of signal regularity, complexity, or ensemble orderliness for time-series data 34 . Reduced complexity in terms of sample entropy indicates that time-series data of joint angles and angular velocities are relatively more regular, less random, and less complex. Inter-and intra-limb synchronization of joint angles and joint angular velocities were quantified using Pearson correlation coefficients. These are measures of dependence between two paired data points with linear association and were used as a similarity index. It can reflect simultaneous changes of joint angles or joint angular velocities among bilateral upper and lower limbs: the joint angles and joint angular velocities of shoulders, elbows, hips, and knees at the right and left sides.
Hammersmith infant neurological examination. The HINE was utilized to assess neurological outcomes in infants at 4 months of corrected age. The HINE was evaluated by an experienced physiatrist or physical therapist and was blinded to the infants' medical histories. The HINE comprises five sections of examinations including assessments of cranial nerve function, posture, movements, tone, and reflexes and reactions with scores ranging from 0 to 78. The five sections were evaluated separately and then added to obtain the global scores. The global score at the corrected age of 3-4 months ranges from 62.5 to 69 35,36 . Global scores < 56 at 3 months of corrected age are considered highly indicative of cerebral palsy, and it is widely accepted that HINE scores of 40-60 at 3-6 months of corrected age indicate gross motor function classification system scores I-II 1,37 . Considering this information, the study participants were divided into the following two groups according to the HINE scores: HINE < 60 and ≥ 60.
Statistical methods. The clinical characteristics were compared between infants with HINE < 60 and those with HINE ≥ 60 using Mann-Whitney U test for continuous variables and chi-squared test or Fisher's exact test for categorical variables. Pearson correlation coefficients were obtained to analyze the correlations between HINE scores and complexity indices for joint angles and joint angular velocities. The complexity and similarity indices for joint angles and joint angular velocities were compared between the HINE < 60 and ≥ 60 groups using

Results
Clinical characteristics. In total, 65 infants who fulfilled the inclusion criteria were enrolled. Table 1 lists the clinical characteristics of 16 infants with HINE < 60 and 49 with HINE ≥ 60. There were significant differences between the two groups in the incidence of multiple gestations (p = 0.002) and patent ductus arteriosus (p = 0.040).

Descriptive statistics of spontaneous movements. Illustrations of joint angles and joint angular
velocities at the bilateral shoulders, elbows, hips, and knees are shown in Fig. 2 Table 2). The complexity indices of joint angles and  www.nature.com/scientificreports/ angular velocities differed significantly between the infants with HINE < 60 and ≥ 60 in most of the joints at the upper and lower extremities (Table 3). Joint angles and angular velocities indicated very weak or weak correlations in most of the joints at the upper and lower extremities, except for the ipsilateral hip and knee joints, which showed strong or very strong correlations (Supplementary Tables S3 and S4

Discussion
The current study demonstrated that the complexity of term-equivalent spontaneous movements in preterm infants quantitatively analyzed using a deep learning algorithm is associated with early neurological development assessed by HINE at 4 months of corrected age 38,39 . The complexity indices of both joint angles and joint angular velocities were different between the very preterm infants with HINE < 60 and ≥ 60, and showed positive correlations with the HINE scores in most joints of the upper and lower extremities. The similarity indices among each joint angle or among each joint angular velocity did not differ between the infants with HINE < 60 and ≥ 60 in most of the joints at the upper and lower extremities.
In this study, reduced complexity of spontaneous movements was significant or demonstrated a trend towards significance for all of the upper and lower extremities in the preterm infants with early neurological development. Global scores of HINE at 3-6 months of corrected age are significantly associated with levels of Gross Motor Function Classification System in cerebral palsy and cognition at 2 years of age in previous studies 38,39 . The complexity indices of both joint angles and joint angular velocities showed a significant positive linear relationship with the global HINE scores even though the strength of relationships was weak. These results are overall consistent with those of previous reports that describe decreased complexity as one of the important findings in infants at a high risk of cerebral palsy 36,40,41 . Hypothetically, spontaneous movements are endogenously generated by the central pattern generator network in the spinal cord and brainstem 6 . Its activity may result in simple body movements but can be modulated by the activity of the supraspinal structures (e.g. subplate and cortical plate) during brain development; this may induce movement complexity 40 . One of the characteristics in deviant patterns of the spontaneous movements is a lack of variability; this can result from injuries or dysfunction of the subplate or cortical plate and/or its connective fibres 41 .
Complexity, which comprises spatial and temporal variability, was investigated to objectively measure spontaneous movements in preterm infants since reduced complexity is strongly associated with cerebral palsy 40,41 . SE was adopted as a key index to measure complexity quantitatively. Generally, reduced complexity indicates fetal or neonatal compromise and has been regarded as one of the important characteristics to distinguish normal and abnormal spontaneous movements during the writhing period 42,43 . A previous study demonstrated that SE of time-series kinematic data acquired from infants with inertial sensors attached to the lower extremities is associated with a risk of developmental delay 33 . In that study, SE of the lower-limb movements was significantly www.nature.com/scientificreports/ decreased in the infants at risk of developmental delay, which is consistent with the results of the current study. Conversely, similarity in terms of the Pearson correlation coefficients did not correspond with the results of complexity. It may be assumed that the similarity of the inter-or intra-limb movements reflects the characteristics of cramped-synchronized movements, but the similarity index, in terms of Pearson's correlation coefficients, may be insufficient to discriminate infants according to the early neurological development. It is necessary to plan future studies that reveal the quantitative features of each type of spontaneous movements according to GMA. Automatic computer-based analyses of spontaneous infantile movements have been explored for clinical use in numerous previous studies 16,17 . One of the main approaches applied is a two-dimensional video-based approach. This has been strengthened by the rapid progress of computer vision technologies, such as deep learning algorithms, and the popularization of smartphone cameras 19 . In the current study, AlphaPose, an open-source pose-estimation model that demonstrates state-of-the-art performance, was used to extract the positional coordinates of each joint at the upper and lower extremities 27 . The acquisition of positional coordinates was conducted automatically using AlphaPose, facilitating the efficient performance of kinematic analyses of spontaneous infantile movements. The pose-estimation model provides automatic acquisition of spatiotemporal data on the trunk, arms, and legs of infants without labour-intensive and time-consuming manual marking. This study determined the kinematic parameters as joint angles and joint angular velocities, rather than as the raw data of the positional coordinates. Joint angles were considered beneficial since they can be less affected by the camera's height, tilt/roll angles, and directions compared to positional coordinates and be straightforward for clinicians to analyze [44][45][46] .
This study has several limitations. First, in this study, a neurological outcome in infants was the global score of HINE at 4 months of corrected age. Even though a low global score of HINE at an early stage is suggestive of developmental delay or cerebral palsy, future studies are necessary to identify the long-term longitudinal association between complexity indices of spontaneous infantile movements and poor neurological outcomes. Second, the sample size of this study was relatively small and a further study is warranted with a large population of preterm infants. Third, kinematic analyses were performed on the two-dimensional video images, not on the three-dimensional image analyses of spontaneous movements in preterm infants. To obtain easy access to video images of the infants after hospital discharge, this study adopted the image acquisition instruments of conventional or smartphone RGB cameras.
In conclusion, quantitative assessments of spontaneous movements in preterm infants using a deep learning algorithm and sample entropy are feasible. Complexity indices in terms of SE of joint angles and joint angular velocities at both the upper and lower extremities are associated with early neurological development. This study indicates that complexity indices of spontaneous movements can be potential candidates to detect cerebral palsy in high-risk infants. Further studies are warranted with larger cohorts of preterm infants and a focus on developing automatic computer-based models using complexity indices for predicting developmental outcomes in clinical practice.

Data availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.